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INTEGRATED  MULTIMODAL  COMMUNICATIONS  MANAGEMENT  FOR 
AIRBORNE  COMMAND  AND  CONTROL 


Dianne  K.  Popik,  Victor  S.  Finomore  and  Douglas  S.  Brungart 
711  Human  Performance  Wing,  Air  Force  Research  Laboratory 
Wright  Patterson  AFB,  Ohio 

Increasing  the  communication  efficiency  and  accuracy  associated  with  Command  and  Control 
(C2)  operations  is  crucial  in  many  aerospace  applications.  This  communication  intensive 
environment  imposes  a  high  workload  on  Air  Traffic  Controllers  and  other  C2  personnel  who 
rely  heavily  on  a  variety  of  communication  tools  to  efficiently  plan,  direct,  coordinate,  and 
control  assets  during  missions.  The  C2  task  is  further  complicated  by  the  suboptimal  integration 
of  the  various  communication  media  utilized  in  the  operational  environment.  The  fielded 
communication  tool  suites  have  serious  limitations  and  are  not  poised  to  meet  the  needs  of 
advancing  technology.  A  multidisciplinary  research  team  in  the  Battlespace  Acoustics  Branch  of 
the  Air  Force  Research  Laboratory  has  developed  the  Multi-Modal  Communications  (MMC) 
tool  suite  which  is  specifically  designed  to  increase  communication  effectiveness,  provide 
efficient  voice  communication  retrieval,  navigation  of  saved  data  as  well  as  reducing  the 
perceived  mental  workload  of  the  operators. 


Challenges  in  Command  and  Control  Communications 

Command  and  Control  communications  are  challenging  for  many  reasons.  This  time  critical 
communication  intensive  environment  imposes  a  high  workload  on  the  operators,  who  typically  monitor 
and  transmit  on  as  many  as  eight  or  more  simultaneous  voice  channels  in  addition  to  other  forms  of 
communication  such  as  text  chat,  phone  and  email.  Reducing  the  workload  and  increasing  the 
communication  efficiency  associated  with  C2  operations  is  of  extreme  importance  to  achieve  mission 
success. 


Voice  communications  pose  a  unique  problem  due  to  the  transient  nature  of  verbal 
transmissions.  The  recipient  has  one  chance  to  extract  the  crucial  information  or  be  forced  to  request  a 
“repeat”  which  adds  to  the  radio  traffic.  Additionally,  multiple  operators  can  speak  at  the  same  time  thus 
reducing  the  intelligibility  of  essential  messages.  Not  only  is  missed  communication  a  problem  during 
the  mission,  but  the  operators  must  take  detailed  notes  to  capture  the  information  received.  Later,  they 
must  rely  on  manually  transcribed  audio  recordings  of  the  mission  for  training  and  debriefing. 

Currently,  C2  centers  are  not  required  to  standardize  the  collaboration  tools  available  and  used 
by  the  operators.  Therefore,  it  is  possible  to  have  very  different  collaboration  tools  in  various  C2 
facilities.  Since  multiple  collaboration  tools  are  needed  to  accomplish  the  mission,  combinations  of 
these  collaboration  tools  are  often  kludged  systems  which  are  meant  to  assist  the  operator  with  the 
management  of  these  individual  systems.  The  incompatibility  of  these  multiple  modes  of 
communication  coupled  with  the  variable  combination  of  systems  makes  it  difficult  to  combine  them 
into  a  functioning  collaborative  tool  suite  suitable  for  all  C2  environments.  Presently,  the  integration  of 
existing  communication  technologies  has  not  yielded  optimal  results.  The  fielded  communication  tool 
suites  have  serious  limitations  and  are  not  poised  to  meet  the  needs  of  advancing  technology. 

In  addition  to  the  lack  of  standardization  and  the  incompatibility  of  the  communications  systems, 
there  currently  is  very  little  capacity  to  capture,  save  and  review  these  various  communications.  For 
instance,  verbal  radio  communications  are  transient  and  perishable  and  reviewing  verbal 
communications  during  a  mission  is  impossible  at  the  present  time.  A  multidisciplinary  research  team  in 
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the  Battlespace  Acoustics  Branch  of  the  Air  Force  Research  Laboratory  has  developed  the  Multi-Modal 
Communications  (MMC)  tool  suite  which  is  specifically  designed  to  address  these  issues. 


Current  Collaborative  Tools 

Operators  surveyed  (Berry  et  al.,  2006)  in  various  C2  centers  listed  the  collaborative  tools  most 
commonly  used  as  phone,  Chat/text  messaging  (mlRC),  email,  radio,  secure  telephone  unit/secure 
terminal  equipment  (STU/STE),  Voice  over  Internet  Protocol  (VoIP),  Information  Work  Space  (1WS), 
and  video  teleconference  (VTC).  Operators  admitted  there  were  too  many  different  collaboration  tools 
available  yet  each  was  limited  in  its  own  way.  Operators  stated  that  phone,  text  chat  and  VoIP  were 
somewhat  effective  pending  their  availability.  Of  these  collaborative  tools,  most  operators  preferred  text 
chat.  Perhaps  this  preference  is  due  in  part  to  the  poor  audio  tools  available  making  them  an  undesirable 
form  of  communication  or  it  may  be  due  to  the  convenience  of  a  written  record  of  communications  that 
text  chat  provides  coupled  with  the  overall  system  stability  of  the  chat  functions  which  were  less  likely 
to  crash. 

Information  Work  Space 

Information  Work  Space  combines  a  few  collaborative  tools,  but  has  limitations  in  sharing, 
posting,  accessing,  filing  and  attaching  documents  as  well  as  data  loss  when  1WS  frequently  crashes 
(Berry  et  al.,  2006).  The  resulting  work  following  a  crash  entails  rebuilding  chat  rooms  and  regaining 
situation  awareness  (SA).  Another  drawback  of  1WS  is  its  limited  audio  capability  due  to  the  simplex 
system  similar  to  a  walkie-talkie  where  one  party  speaks  at  a  time.  A  simplex  system  limits  the  ability  to 
communicate  naturally  by  prohibiting  interjections  and  not  surprisingly  is  an  unpopular  form  of 
communication.  1WS  chat  functions  were  more  often  used  than  voice  functions  with  operators  using  on 
average  five  chat  windows  at  a  time,  while  a  few  operators  had  as  many  as  14  chat  windows  open. 

Text  Messaging/Chat 

Text  Messaging/Chat  was  commonly  used  especially  when  1WS  was  unavailable  (Berry  et  al., 
2006).  When  this  occurred  the  operators  would  switch  to  mlRC  which  is  an  Internet  Relay  Chat  (IRC) 
client  for  Microsoft  Windows.  Despite  mlRC’s  popularity,  the  operators  noted  it  is  difficult  to 
reconfigure  following  a  system  crash.  In  this  setting,  mlRC  was  used  only  temporarily  and  once  1WS 
was  returned  to  functioning  status,  mlRC  communications  were  duplicated  and  placed  into  1WS  again. 

A  common  complaint  about  the  chat  tool  was  the  inability  to  copy  large  amounts  of  data  into  a  chat 
room.  Brief  departures  from  the  chat  room  also  caused  a  loss  of  SA  since  it  was  not  possible  to  tag  the 
last  entry  read.  The  operators  would  be  forced  to  spend  precious  time  re-reading  text  to  ascertain  where 
they  previously  left  the  stream  of  communication. 

Other  Collaborative  Tools 

Email  offered  a  limited  capacity  in  such  a  dynamic  and  high  tempo  environment  (Berry  & 
Lindberg,  2009).  Phone  and  STU/STE  communication  was  inconvenient  since  it  required  operators  to 
remove  their  headset  in  order  to  hold  the  phone  to  the  ear.  VoIP  functioned  similarly  to  telephone  use  but 
heard  through  the  headset.  VTC  was  available  in  certain  facilities  in  the  building  and  not  at  individual 
terminals.  Radio  was  often  used;  however,  the  operators  rarely  wore  both  ear  cups  in  order  to  hear 
conversation  around  them. 
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Purpose  of  Multi-Modal  Communications  Tool  Suite 

The  Multi-Modal  Communications  (MMC)  tool  suite  was  designed  to  offer  C2  operators  a 
combined  versatile  and  intuitive  interface  which  would  alleviate  the  workload  and  errors  associated  with 
an  intensive  communication  environment.  This  integrated  Communications  Management  tool  will 
improve  communications  by  streamlining  the  cumbersome  and  varied  forms  of  communications  and 
give  the  C2  operator  access  to  the  complete  spectrum  of  communications  in  a  single  tool.  Voice  and 
chat  communications  will  be  seamlessly  integrated  into  a  single  digital  communication  system  over  one 
headset  (phone,  chat,  voice,  radio)  for  internal  and  external  communications. 

MMC  is  an  integrated  net-centric  architecture  which  will  distribute,  monitor,  archive,  and 
retrieve  analog  voice  transmissions  (radio  communications),  VoIP  communications,  and  text  messages 
across  distributed  operators  on  the  GIG. 

MMC  records,  archives  and  displays  the  verbal  and  text  communications  to  the  operator  for  real¬ 
time  playback  during  the  missions  reducing  workload  while  enabling  the  C2  operator  to  be  more 
effective  and  efficient.  This  eliminates  the  perishable  nature  of  radio  communication  and  allows  the 
operator  to  focus  on  the  task  instead  of  remembering  and  writing  down  information.  The  MMC  tool  also 
employs  virtual  audio  display  technology  to  spatialize  the  multiple  audio  signals  to  aid  in  the 
intelligibility  of  the  radio  communication.  The  combination  of  these  technologies  has  led  to  the  design  of 
a  communication  interface  that  will  improve  the  performance  of  operators  confronted  with  monitoring  a 
high  volume  of  radio  communication. 

Features  of  Multi-Modal  Communications  Tool  Suite 


Audio  Recording 

The  MMC  tool  captures  the  radio  communication  as  text  and  records  each  transmission  as  an 
audio  file.  Operators  have  the  ability  to  play  back  the  original  radio  transmission  by  clicking  on  the 
desired  line  of  transcribed  text.  Each  audio  file  or  radio  transmission  is  time  stamped  for  easy  reference 
and  documentation. 

Speech-to-Text  Transcription 

The  speech-to-text  transcription  feature  captures  incoming  speech  and  transcribes  it  into  text. 
This  allows  all  voice  traffic  to  be  captured  and  recorded  as  a  text  log.  The  operator  now  has  the  ability 
to  read  what  was  spoken  and  review  previous  voice  transmissions.  Since  all  radio  communications  are 
logged,  the  operator  is  easily  able  to  search  for  keywords  during  the  mission  or  use  the  text  for 
debriefing  or  training  purposes. 

Spatial  Audio 

The  spatial  audio  feature  (also  called  3D  Audio)  allows  users  to  spatialize  each  of  their 
monitored  radio  channels  such  that  the  audio  signals  appear  to  originate  from  different  azimuth 
locations.  MMC  allows  the  operator  to  place  the  radio  channels  in  one  of  nine  spatial  locations  and  to 
change  that  location  anytime  during  the  mission.  This  flexibility  of  configuration  allows  the  operator  to 
organize  and  more  efficiently  monitor  multiple  radio  channels.  Several  studies  have  shown  that  the 
spatialization  of  speech  can  improve  the  intelligibility  of  communication,  lower  the  perceived  mental 
workload  associated  with  monitoring  simultaneous  streams  of  communication,  and  decrease  the  negative 
effects  of  noise  during  communication  (Bolia,  2003;  McAnally,  Bolia,  Martin,  Eberle,  &  Brungart,  2002; 
Nelson,  Bolia,  Ericson,  &  McKinley,  1998;  Nelson,  et  all 999;  Ricard  &  Meirs,  1994). 
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Instant  Replay 

The  instant  replay  feature  provides  immediate  access  to  the  last  message  transmitted.  By 
pressing  the  ‘replay’  button,  the  last  fifteen  seconds  of  the  radio  transmission  will  be  replayed,  allowing 
the  operator  to  instantly  review  or  clarify  the  last  transmission.  Additionally,  the  replay  feature  isolates 
that  particular  channel  by  temporarily  muting  all  other  channels. 

Isolate 


The  isolate  feature  mutes  all  other  radio  channels  and  only  plays  the  specified  channel,  allowing 
the  operator  to  focus  their  attention  solely  on  that  channel.  The  operator  may  now  more  effectively 
direct  their  attention  to  critical  situations  by  muting  less  critical  radio  transmissions. 

Transmit 

The  push-to-talk  feature  allows  the  operator  to  speak  and  be  heard  by  other  operators  monitoring 
the  specified  channel.  The  push-to-talk  is  activated  by  mouse  clicking  the  ‘Talk’  button  and  holding  it 
down  while  speaking.  Other  operators  listening  to  that  channel,  hear  the  communication  in  real-time 
while  the  spoken  communication  is  transcribed  into  the  display  window. 


Figure  1.  Multi-Modal  Communications  tool  suite  display  of  a  single  radio  communications  window. 


Future  Capabilities 

Since  the  operators  prefer  the  use  of  chat  messaging  while  communicating;  chat  capabilities  are 
currently  being  implemented  into  the  MMC.  This  chat  feature  will  be  similar  to  current  chat  clients  in 
that  users  will  sign  into  secured  chat  rooms  to  monitor  and  transmit  text  messages.  This  chat  function 
will  have  the  capacity  to  convert  text  to  speech.  Operators  will  have  the  option  to  speak  their  messages  to 
be  transcribed  in  the  chat  window  as  well  as  hear  text  messages  in  a  synthesized  voice.  Additional 
features  will  enable  the  operator  to  review  logged  communications  in  faster  than  real-time  to  be  able  to 
review  large  chucks  of  past  communication  in  a  speeded  form  in  order  to  be  brought  up-to-date. 


4 

Data  subject  to  restrictions  on  Cover  and  Notice  Page 
Approved  for  public  release;  Distribution  is  unlimited 


Research  and  Survey  Data 


Throughout  the  development  of  the  MMC  tool,  1 1  subject  matter  experts  (SME)  were  shown 
MMC  and  asked  to  participate  in  a  usability  questionnaire  and  interview.  The  SMEs  had  diverse  C2 
operations  experience  ranging  from  1  to  27  years  (M=  1 1.57),  thus  providing  comprehensive  feedback 
on  the  features  and  design  of  MMC.  This  feedback  led  to  the  redesign  of  the  interface  making  it  more 
operator-friendly  and  congruent  with  the  operator’s  needs.  The  SMEs  indicated  that  MMC  would 
improve  job  performance  by  allowing  them  to  accomplish  tasks  more  quickly  and  efficiently.  MMC  was 
unanimously  supported  by  all  of  the  SMEs  involved  as  an  essential  tool  in  the  field  and  a  valuable  tool 
for  training  and  debriefing.  Feedback  also  included  comments  indicating  the  spatial  audio  feature  would 
enhance  talker  identification  and  speech  intelligibility  while  the  transcription  and  playback  features 
would  help  reduce  miscommunications  and  the  need  for  call  backs.  Overall  the  SMEs  signified  that 
MMC  was  intuitive  and  easy  to  use. 

In  addition  to  collecting  questionnaire  data,  performance  measures  were  also  collected  to 
compare  operators’  ability  to  detect  and  respond  to  critical  messages  via  standard  radio  communication 
alone  (no  3D  audio,  no  voice-to-text  and  no  replay)  and  with  the  MMC  (3D  audio,  voice-to-text,  and 
replay).  Operators  monitored  six  radio  channels  for  ten  minutes  for  the  presence  of  a  critical  message 
which  identified  a  hostile  entity  along  with  information  pertaining  to  their  location  (Viper  1,  Hostile- 
North  Lead  Group,  55  miles).  Their  task  was  to  repeat  that  information  back  on  the  correct  radio 
channel.  There  were  six  radio  transmissions  per  min  with  the  occurrence  of  one  critical  message  per  min 
on  each  of  the  channels.  Nine  paid  participants  from  the  General  Dynamics  research  pool  at  Wright- 
Patterson  Air  Force  Base  in  the  Battlespace  Acoustics  Branch  served  as  participants  in  this  study.  Three 
separate  2  (Condition)  x  2  (Trails)  Within- Analysis  of  Variance  was  performed  for  measures  of  Correct 
Detection,  Response  Accuracy,  and  Response  Time.  The  data  revealed  that  participants  were  better  at 
detecting  critical  messages  in  the  MMC  condition  ( MCD  =  72.  31%)  then  in  the  radio  condition  (MCd  = 
50.18%),  F(l,  8)  =  23.23, p<  .01.  The  ANOVA  on  the  response  accuracy  also  showed  that  MMC 
condition  (MA  =  94.27%)  was  greater  than  the  radio  condition  (MA  =  82.56%),  F(l,  8)  =  7.20,  p  <  .05. 
Analysis  of  response  time  revealed  that  it  took  longer  to  reply  when  using  MMC  (MT  =  1 1 .59  s)  then  the 
standard  radio  (MT  =  7.65  s),  F(l,  8)  =  5.39,  p  <  .05.  Thus  it  seems  the  addition  of  the  voice-to-text 
capability  in  the  MMC  algorithm  improves  overall  performance  in  detection  and  response  accuracy  but 
may  increase  response  time,  presumable  because  listeners  who  are  uncertain  were  waiting  for  the 
transcribed  text  before  making  a  response.  It  is  important  to  note  that  there  was  on  average  a  5  to  8  sec 
delay  in  the  voice-to-text  transcription  thus  inherently  increasing  the  time  it  took  participants  relying  on 
the  text  to  reply.  The  transcription  latency  has  since  been  decreased  to  less  than  a  second  thus  continuing 
investigation  of  the  performance  of  the  MMC  is  in  progress.  None  of  the  other  main  effect  of 
interactions  in  these  analyses  reached  significance,  p  >.05  in  all  cases. 

Conclusions 

MMC  has  integrated  several  stand-alone  features  known  to  improve  communications  in  order  to 
create  a  network-centric  communication  management  suite.  The  data  collection  and  feedback  from 
subject  matter  experts  (SME)  indicates  that  MMC  has  the  potential  to  improve  the  communication 
effectiveness  of  operators  in  intense  communication  environments.  Although  MMC  currently  does  not 
meet  all  of  the  needs  of  an  operator  we  fully  expect  the  empirical  tests,  both  in  the  lab  and  field  studies, 
to  show  that  MMC  does  in  fact  improve  performance  during  communication  monitoring  tasks.  We  also 
expect  these  tests  to  highlight  other  features  to  further  develop  in  the  MMC  thus  creating  a  fully 
functional  multi-modal  communication  suite. 
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